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Key findings 



Students who drop out of high school are at increased risk of a range of negative social and 
economic consequences. School districts are using early warning indicators, such as attendance, 
course failures, grade point average, and suspensions or expulsions, to identify and provide 
supports for students at risk of dropping out. This study in six Washington state districts 
examined whether two commonly used early warning indicators work equally well for English 
learner students. It found that: 

Students who had ever been English learners had four-year graduation rates that were 
9.5 percentage points lower and dropout rates that were 0.7 percentage point higher than 
those of students who had never been English learners. 

Certain subgroups of English learner students had considerably different graduation and 
dropout rates from those of other English learner students; relative to long-term proficient 
English learner students, newcomer English learner students’ four-year graduation rates were 
33.8 percentage points lower and their dropout rates were 5.8 percentage points higher. 
The early warning indicators used by the districts were unable to accurately identify many 
future dropouts across the six districts, especially among newcomer English learner students. 
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Summary 


Students who drop out of high school are at increased risk of a range of negative social 
and economic consequences, including lower earnings and poorer health (Pleis, Ward, & 
Lucas, 2010; Rouse, 2007). To reduce dropout rates and lessen these negative consequences, 
districts around the country are using early warning indicators to identify and provide sup- 
ports for students at risk of dropping out. Typically, these early warning indicators include 
some combination of attendance, course failures, grade point average, and suspensions or 
expulsions (Allensworth & Easton, 2007; Hartman, Wilkins, Gregory, Gould, & D’Souza, 
2011; Silver, Saunders, & Zarate, 2008). 

It is not clear whether these commonly used early warning indicators work equally well 
for English learner students. National data suggest that English learner students drop out 
of high school at higher rates than other students do (Kena et al, 2015; U.S. Department 
of Education, 2015). English learner students are a heterogeneous group that includes stm 
dents born in the United States, new immigrants, and refugees, all with varying degrees 
of prior exposure to English and, for those not born in the United States, different edm 
cation experiences prior to arrival. Some English learner students start kindergarten in 
this country, while others do not arrive here until they are in high school. Some English 
learner students may receive only two or three years of English language services, while 
others may receive eight or more years. Current national data do not capture the variation 
in dropout and graduation rates for those different types of English learner students. 

This study compares data for a particular group of students — those who were classified 
as English learner students at any point in their K-12 education (referred to as “ever- 
English learner students” in this report) with data for students who were never classified 
as English learner students (referred to as “never-English learner students”). It also com- 
pares outcomes across subgroups of the ever-English learner student population. Specifi' 
cally, the study addresses how the graduation and dropout rates of different subgroups of 
ever-English learner students compare with one another and with those of never-English 
learner students. And it examines whether two early warning indicators used to predict 
dropping out (six or more absences in grade 9 plus at least one course failure in grade 9, 
and at least one suspension or expulsion in grade 9) are accurate and useful indicators 
for different groups of ever-English learner students compared with never-English learner 
students. 

The students in the study are from six school districts in the south King County area 
of Washington state. The districts are part of the Road Map Project, a cradle-to-career 
initiative that seeks to double the number of students on track to graduate from college 
or earn a career credential between 2010 and 2020. As part of the initiative, the districts 
have been using a common set of early warning indicators since 2011. The initiative also 
has a work group focused on the large number of English learner students in the region. 
Regional Educational Laboratory Northwest has partnered with the work group since 2012 
to use data and evidence to better understand the needs and challenges of English learner 
students and to inform decisions about policy and practice. 


The study findings highlight notable differences in graduation and dropout rates among 
subgroups of English learner students. Key findings from the study include: 

• Students who had ever been English learners had founyear graduation rates that 
were 9.5 percentage points lower and dropout rates that were 0.7 percentage point 
higher than those of students who had never been English learners. 

• Certain subgroups of English learner students had considerably different gradm 
ation and dropout rates from those of other English learner students; relative to 
long-term proficient English learner students, newcomer English learner students’ 
four-year graduation rates were 33.8 percentage points lower and their dropout 
rates were 5.8 percentage points higher. 

• The Road Map Project’s early warning indicators, originally developed for Seattle 
Public Schools, were unable to accurately identify many future dropouts, especially 
among newcomer English learner students. 

Given that the accuracy of the Road Map Project indicators varied for subgroups of English 
learner students and may be evidence of the need to select and validate indicators specif- 
ically for the population of interest, the Road Map Project and other states and districts 
may want to examine the accuracy of their own indicators for different student popula- 
tions. If early warning indicators are weaker for a specific subgroup of English learner stu- 
dents, then teachers, counselors, and others may want to monitor the needs of that group 
in other ways. 


Contents 


Summary i 

Why this study? 1 

What the study examined 3 

What the study found 5 

Students who had ever been English learners had graduation rates that were 9.5 percentage 
points lower and dropout rates that were 0.7 percentage point higher than those of 
students who had never been English learners 5 

Certain subgroups of English learner students had considerably different graduation and 
dropout rates from those of other English learner students; relative to long-term 
proficient English learner students, newcomer English learner students’ four-year 
graduation rates were 33.8 percentage points lower and their dropout rates were 
5.8 percentage points higher 6 

The Road Map Project early warning indicators were unable to accurately identify many 

future dropouts across the six districts, especially among newcomer English learner students 8 

Implications of the study findings 10 

Limitations of the study 12 

Appendix A. Data and methodology A-1 

Appendix B. Supplemental results B-1 

References Ref-1 

Boxes 

1 Definitions of English learner student subgroups 3 

2 Data and methods 4 

Tables 

1 Number and percentage of students in six Washington state districts who graduated 

within four or five years of entering grade 9 in 2008/09, by English learner status 6 

2 Number and percentage of students in six Washington state districts who dropped out 

within four years of entering grade 9 in 2008/09, by English learner status 7 

3 Number and percentage of students in six Washington state districts who neither 

graduated nor dropped out after four or five years of entering grade 9 in 2008/09, by 
English learner status 8 

4 Percentage of students in six Washington state districts who triggered each early 
warning indicator, for all students and English learner student subgroups who entered 

grade 9 in 2008/09 9 

5 Odds ratios, precision, and sensitivity of the composite dropout indicator for students in 

six Washington state districts who entered grade 9 in 2008/09, by English learner status 10 
A1 Example of classification of results from logistic regression A-5 


B1 Odds ratios for early warning indicators as predictors of dropping out for students from 

six Washington state districts who entered grade 9 in 2008/09, by English learner status B-l 
B2 Summary of classification accuracy results by indicator and overall for students from six 

Washington state districts who entered grade 9 in 2008/09, by English learner status B'2 


IV 


Why this study? 


Students who drop out of high school are at increased risk of a number of negative social 
and economic consequences, including lower earnings and poorer health (Pleis et al, 2010; 
Rouse, 2007). To reduce dropout rates, districts around the country have adopted early 
warning indicators to identify students at risk of dropping out of high school (for example, 
Allensworth & Easton, 2007; Callahan, 2013; Hartman et al., 2011). Identified students 
may receive tailored interventions designed to keep them in high school and help them 
earn a high school diploma. Typically, early warning indicators include some combina' 
tion of attendance, course failures, grade point average, and suspensions or expulsions 
(Allensworth & Easton, 2007; Hartman et al., 2011; Silver et al., 2008). 


Early warning indicators developed using data from the general student population may not 
be as helpful for identifying the specific risk factors for English learner students. National 
data for 2013/14 reveal a nearly 20 percentage point gap between the overall graduation 
rate (82.3 percent) and the rate for English learner students (62.6 percent), although the 
size of the gap varies across states (U.S. Department of Education, 2015). Limited national 
evidence about dropout rates among English learner students suggests that these students 
drop out of high school at higher rates than other students do. Eor example, in 2012 a 
quarter of Hispanic students born outside the United States dropped out of high school, 
compared with 6 percent of all students born in the United States (Kena et al, 2015). 


National statistics on English learner students’ education outcomes may also obscure the 
full range of this heterogeneous group’s experiences. The group includes students who were 
born in the United States, as well as immigrants from around the world with a wide range 
of linguistic and academic skills. Eurthermore, unlike racial/ethnic subgroups, English 
learner status is not a fixed characteristic — after attaining proficiency in English, these 
students are reclassified as former English learner students. All former English learner stm 
dents may not have the same level of English proficiency because reclassification criteria 
vary from state to state and sometimes even within a state. In addition, because some 
school and district policies limit English learner students’ access to core and advanced 
courses, the grade level in which students are reclassified as former English learner students 
may substantially alter students’ academic experiences (Estrada, 2015; Umansky, 2015). 

This study examined four- and five-year high school graduation rates and four-year dropout 
rates for different subgroups of current and former English learner students. It examined 
both graduation and dropout rates because not all students who fail to graduate are drop- 
outs. Some students remain enrolled in school, and for other students, end-of-high school 
outcomes are unknown. The study also examined the accuracy of early warning indicators 
of dropout because that is the outcome most indicator systems are designed to predict. (See 
appendix A for a description of graduation and dropout rates and the accuracy of early 
warning indicators.) 

The context for the study is the Regional Educational Laboratory Northwest’s research 
alliance with the Road Map Project, a collective impact initiative of seven school dis- 
tricts in the south King County area of Washington state. These seven districts — Auburn, 
Eederal Way, Highline, Kent, Renton, Tukwila, and the southern part of the Seattle 
Public Schools district — serve a highly diverse population of more than 150,000 students, 
19 percent of whom are current English learner students and many more of whom are 
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former English learner students (Community Center for Education Results, 2015a). The 
analysis covers data for only six of these districts because one district (Highline) did not 
provide data for this study. 

Early identification of English learner students at risk of dropout is crucial for the Road 
Map Project to achieve its goal of doubling the number of students who are on track to 
graduate from college or earn a career credential by 2020, compared with the number in 
2010 (Community Center for Education Results, 2015b). The initiative supports districts by 
monitoring and reporting how many grade 9 students are not on track to graduate. When 
the Road Map Project launched in 2010, some of the districts involved had not yet fully 
developed their own early warning indicator systems, although many had partial systems 
in place (Cunningham & Van Alstyne, 2012). Seattle Public Schools, the largest of the 
Road Map Project districts, had adopted two early warning indicators in 2009, informed in 
part by analyses of the district’s dropout population (Celio, 2009a, 2009b, 2011). Students 
were identified as at risk of dropout if they reached the threshold for either indicator: six 
or more absences plus at least one course failure in grade 9 (indicator 1) or at least one 
suspension or expulsion in grade 9 (indicator 2). 

Searching for common measures to use across districts, the Road Map Project adopted 
these indicators and currently uses them in its annual reporting on progress across districts 
and subgroups (Community Center for Education Results, 2011). In 2015 the initiative 
began comparing the percentages of current English learner students and non-English 
learner students (both former English learner students and never-English learner students) 
who trigger an indicator, but these reports do not distinguish results for former English 
learner students or different groups of current English learner students (Community 
Center for Education Results, 2015a). 


This study aims to assist Road Map Project stakeholders in two ways, hirst, it compares 
graduation and dropout rates among a diverse range of English learner students: students 
entering grade 9 as recent U.S. arrivals, students entering high school who had been clas- 
sified for several years as English learners, students reclassified as former English learners 
in elementary school, and students reclassified more recently. The findings may help the 
Road Map Project districts and others gain a better understanding of which students are 
at greatest risk of not graduating and therefore may need additional supports or alterna' 
tive monitoring. Second, the study assesses the accuracy of the initiative’s early warning 
indicators, with an emphasis on the indicators’ ability to accurately predict dropout among 
English learner students. It also examines accuracy for the different subgroups of English 
learner students. 

The study may also be useful to several national audiences. Eor state and local agencies 
and collective impact initiatives that are implementing early warning indicator systems, 
the study raises an important question: Are the types of indicators commonly used appro- 
priate and accurate for all student populations? Eor educators working with English learner 
students, the study calls attention to the different experiences of subgroups within the 
larger category of students who were ever classified as English learner students. 
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What the study examined 


This report presents the results of two sets of analyses. First, it compares graduation and 
dropout rates between students who were classified as English learner students at any point 
in their K-12 education (ever-English learner students) and students who were never clas- 
sified as English learner students (never-English learner students), as well as four subgroups 
within the sample of ever-English learner students. (These subgroups are defined in box 1; 
more information about the study data and methods is in box 2 and appendix A.) Infon 
mation about graduation and dropout rates provides context for understanding the second 
set of analyses, which examine the accuracy of the early warning indicators the Road Map 
Project districts use to assess students’ risk of dropping out. 


Box 1. Definitions of English learner student subgroups 

Ever-English learner students are typically grouped into two categories: current and former English learner students. 

Current English learner students have not yet achieved proficiency in English by grade 9. Current English learner stu- 
dents are further divided into two subgroups that reflect distinctions made in the research literature and correspond 
to practitioners’ conceptualization of important differences among English learner students: 

• Newcomer English learner students (or “newcomers”) are recent immigrants who first entered a school in Wash- 
ington state in grades 7, 8, or 9 and were still classified as current English learner students in grade 9. 

• Established English learner students were first classified as English learner students in grade 6 or earlier and 
were still classified as English learner students in grade 9. 

Former English learner students have achieved proficiency, as indicated by receiving a score at the highest level 
(level 4) on the state’s English language proficiency assessment, which reclassifies them as former English learner 
students. Former English learner students are also divided into two subgroups: 

• Recently reclassified English learner students are English learner students who reached proficiency and were 
reclassified as former English learner students in grade 7 or grade 8. 

• Long-term proficient English learner students are English learner students who reached proficiency and were 
reclassified as former English learner students in or before grade 6. 

The number of students in each group or subgroup for the six study districts combined are shown in the table. Detailed 
definitions and a description of how students are categorized into student groups and subgroups are in appendix A. 


Sample size and characteristics of student groups and English learner student subgroups in six Washington 
state districts who entered grade 9 in 2008/09 


Student group or subgroup 

Number of 
students 

Grade at entry into 
Washington state schools 

Reclassified 
by grade 7 

Reclassified 
by grade 9 

All students 

9,595 

K-9 

na 

na 

Never-English learner 

6,943 

K-9 

na 

na 

Ever-English learner 

2,652 

K-9 

Varies 

Varies 

Current English learner 

1,333 

na 

na 

na 

Newcomer 

604 

7-9 

na 

No 

Established 

729 

K-6 

No 

No 

Former English learner 

1,319 

na 

na 

na 

Recently reclassified 

216 

K-6 

No 

Yes 

Long-term proficient 

1,103 

K-6 

Yes 

Yes 

na is not applicable. 


Source: Authors’ analysis based on Washington Office of Superintendent of Public Instruction policies and data from six Road Map 
Project districts, 2004/05-2012/13. 
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Box 2. Data and methods 


The Road Map Project school districts and the Washington Office of Superintendent of Public Instruction provided 
data for this study. The Washington Office of Superintendent of Public Instruction provided data from the Compre- 
hensive Education Data and Research System for students enrolled in public schools from 2009/10 to 2012/13. It 
also provided data from the Transitional Bilingual Instructional Program database for English learner students from 
2004/05 to 2012/13. Data included gender, race/ethnicity, grade level, English learner status for each school year, 
credits attempted, credits earned and in which subjects, as well as school and district enrollment and withdrawal 
dates and reasons. Six of the seven Road Map Project school districts provided additional data for the 2006/07- 
2008/09 school years. These data included the same variables that the Washington Office of Superintendent of 
Public Instruction provided for 2009/10-2012/13. Combining data from these sources provided up to eight years 
of data on a total of 9,595 students who started grade 9 in 2008/09 in the districts of Auburn, Federal Way, Kent, 
Renton, Tukwila, and the southern part of the Seattle Public Schools district. 

Calculating graduation and dropout rates 

Understanding how graduation and dropout rates vary provides helpful context because the two statistics are cal- 
culated differently. For each group of students defined in box 1, the study team calculated adjusted graduation 
rates four and five years after students entered grade 9. To calculate the adjusted four-year graduation rate, the 
number of graduates was divided by the number of incoming students in grade 9. Students with withdrawal codes of 
“transferred out" of Road Map Project districts were not included in the calculation of graduation or dropout rates. 
Students who transferred into Road Map Project districts after grade 9 were included in the calculations. Five-year 
graduation rates were calculated using the same formula and an additional year of enrollment records. 

Dropout rates were calculated by dividing the number of students with withdrawal codes of “dropout” or “General 
Educational Development (GED)” at any time within four years of entering grade 9 by the number of incoming stu- 
dents in grade 9, correcting for the number of students transferring in or out of the Road Map Project school dis- 
tricts (the same denominator as that used to calculate the four-year graduation rate). 

Assessing the accuracy of early warning indicators 

The Road Map Project uses two early warning indicators: six or more absences in grade 9 plus at least one course 
failure in grade 9 (indicator 1) and at least one suspension or expulsion in grade 9 (indicator 2). Students are flagged 
as at risk if they trigger either indicator. Descriptive analyses compare how often students in different subgroups 
triggered the indicators. 

To examine the accuracy of early warning indicators, logistic regression models were run for each indicator sep- 
arately and for a composite indicator based on triggering either indicator or both indicators. Results from the logistic 
regression models were reported in odds ratios, which compare the odds (that is, the likelihood) that a group of 
students who triggered either one or both indicators will drop out, compared with the odds that a group of students 
who did not trigger either indicator will drop out. Results provide information about the likelihood that the indicators 
accurately predict dropping out. Models were run separately for all students, never-English learner students, and for 
each English learner student subgroup, allowing for a comparison of the likelihood — and therefore the accuracy — of 
the indicator for different groups of students. 

Two indices are commonly used to describe the accuracy of dropout prediction (Jobs for the Future, 2014): 

• Precision is the percentage of students who actually drop out among all students identified in grade 9 as at risk. 

• Sensitivity is the percentage of students who were identified as at risk of dropping out and who actually drop out 
among all students who actually drop out. 

The main report discusses the precision and sensitivity of the early warning indicators, as they directly pertain 
to the accurate prediction of dropouts. Additional information about other measures of accuracy is in appendix A. 
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Two research questions guided the study: 

• How do the graduation and dropout rates of different subgroups of ever-English 
learner students compare with one another and with those of never-English 
learner students? 

• Are two early warning indicators used to predict dropping out (six or more absenc' 
es in grade 9 plus at least one course failure in grade 9, and at least one suspension 
or expulsion in grade 9) accurate and useful indicators for different groups of ever- 
English learner students compared with never-English learner students? 

What the study found 


This study found differences in graduation and dropout rates between ever- and never- 
English learner students and among subgroups of ever-English learner students. The study 
also found that ever-, never-, and subgroups of ever-English learner students varied in the 
rate at which they triggered the early warning indicators. Einally, the study found that the 
accuracy of indicators varied across subgroups of English learner students. 

Students who had ever been English learners had graduation rates that were 9.5 percentage points 
lower and dropout rates that were 0.7 percentage point higher than those of students who had never 
been English learners 

Graduation rates in the six study districts were lower than national averages. The founyear 
graduation rate in the study districts was 70.8 percent for all students entering grade 9 in 
2008/09. On average, this rate was higher for never-English learner students (73.5 percent) 
than for ever-English learner students (64.0 percent; table 1). This graduation rate for 
never-English learner students is about 6.5 percentage points lower than the national 
average of 80 percent for never-English learner students for 2011/12. The national four- 
year graduation rate for ever-English learner students is not available, but the national 
rate for current English learner students in 2011/12 was 59 percent, which is also about 
6 percentage points higher than the 52.7 percent rate for current English learner students 
in this study (U.S. Department of Education, 2014). 

Eive-year graduation rates are also useful to consider because for some English learner 
students an extra year provides the opportunity to improve their English proficiency and 
accumulate required credits for graduation. Other reasons students continue in high school 
for an additional year include needing extra time to complete an Individualized Educa- 
tion Program (for students receiving special education services), failing to meet graduation 
requirements because of shortcomings in student counseling, mismatch of credits trans- 
ferred from one school with the graduation requirements of the new school, or difficult 
circumstances such as injury or illness that result in interrupted schooling. Never-English 
learner students had higher five-year graduation rates than ever-English learner students 
did, but the gap between the two narrowed from 9.5 percentage points after four years of 
high school to 5.6 percentage points after five years. 

Because graduation rates do not provide adequate information about the outcomes for stu- 
dents who did not graduate after four years of high school, educators also pay attention 
to dropout rates. Some students who do not graduate in four years drop out, while others 
may continue in high school for another year. Although these students may benefit from 
services to help them graduate earlier, they might not benefit from the same interventions 
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Table 1. Number and percentage of students in six Washington state districts 
who graduated within four or five years of entering grade 9 in 2008/09, by English 
learner status 


Student group 
or subgroup 

Total 

number of 
students 

Adjusted four year 
graduation rate 

Number Percent 

Adjusted five year 
graduation rate 

Number Percent 

Change, 
year 4 to 
year 5 
(percentage 
points) 

All students 

9,595 

6,796 

70.8 

7,410 

77.2 

6.4 

Never-English learner 

6,943 

5,100 

73.5 

5,470 

78.8 

5.3 

Ever-English learner 

2,652 

1,696 

64.0 

1,940 

73.2 

9.2 

Current English learner 

1,333 

703 

52.7 

869 

65.2 

12.5 

Newcomer 

604 

257 

42.6 

355 

58.8 

16.2 

Established 

729 

446 

61.2 

514 

70.5 

9.3 

Former English learner 

1,319 

993 

75.3 

1,071 

81.2 

5.9 

Recently reclassified 

216 

150 

69.4 

165 

76.4 

7.0 

Long-term proficient 

1,103 

843 

76.4 

906 

82.1 

5.7 


Note: The adjusted four-year graduation rate is the number of graduates divided by the number of incoming 
students in grade 9, correcting for the number of students transferring in (inciuded) or out (exciuded) of the 
Road Map Project school districts. An additionai year of enroiiment records is inciuded in the five-year adjusted 
graduation rate, which aiso exciudes students who transfer out after the fourth year of high school (about 
10 percent of all students who transfer out). See appendix A for details on the calculation of graduation rates. 

Source: Authors’ analysis of Washington Office of Superintendent of Public Instruction and Road Map Project 
district data, 2004/05-2012/13. 


used to prevent dropping out. Other students end their enrollment in school without a 
reason on record. Because their outcomes after their last year of high school are unknown, 
it is not certain whether dropout prevention interventions could help these students (after 
five years 12.1 percent of students had either missing, “unknown,” or “no show” withdraw- 
al codes; additional information about these students is in table 3 and appendix A). For 
example, it is possible that some of these students moved to another state and graduated 
there but that the transfer was never recorded in the database. A better way to describe 
the population of students who could benefit from dropout prevention interventions is to 
calculate dropout rates that distinguish the percentage of students who withdraw from 
school as a dropout and the percentage who earned a General Educational Development 
(GED) credential in place of a regular diploma. 

After four years of high school the dropout rate for all students who entered grade 9 in 
2008/09 in one of the six districts was 5.4 percent. Ever-English learner students dropped 
out at a higher rate (5.9 percent) than did never-English learner students (5.2 percent; 
table 2). 

Certain subgroups of English learner students had considerably different graduation and dropout 
rates from those of other English learner students; relative to long-term proficient English learner 
students, newcomer English learner students’ four-year graduation rates were 33.8 percentage 
points lower and their dropout rates were 5.8 percentage points higher 

Graduation rates differed substantially among subgroups of ever-English learner students 
(see table 1). Slightly more than half (52.7 percent) of current English learner students 
graduated in four years. Within this group newcomers had a particularly low four-year 
graduation rate (42.6 percent). Former English learner students, in contrast, had a higher 
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Table 2. Number and percentage of students in six Washington state districts who 
dropped out within four years of entering grade 9 in 2008/09, by English learner 
status 


Student group 


Four year dropout rate 

or subgroup 

Total number of students 

Number 

Percent 

All students 

9,595 

520 

5.4 

Never-English learner 

6,943 

364 

5.2 

Ever-Engllsh learner 

2,652 

156 

5.9 

Current English learner 

1,333 

103 

7.7 

Newcomer 

604 

54 

8.9 

Established 

729 

49 

6.7 

Former English learner 

1,319 

53 

4.0 

Recently reclassified 

216 

19 

8.8 

Long-term proficient 

1,103 

34 

3.1 


Note: Graduation rates in tabie 1 and dropout rates in this tabie do not sum to 100 percent because some 
students may have continued in high schooi for a fifth or sixth year, and some students had withdrawai codes 
that did not provide information on whether that student dropped out or transferred to another schooi outside 
the six districts and graduated eisewhere. Such students are inciuded in the denominators of both graduation 
and dropout rates, but not in the numerators. Four- and five-year graduation rates and four-year dropout rates 
are caicuiated using the same denominator. Dropout rates were caicuiated by dividing the number of students 
with withdrawai codes of “dropout” or “Generai Educationai Deveiopment (GED)" at any time within four years 
of entering grade 9 by the number of incoming students in grade 9, correcting for the number of students 
transferring in (inciuded) or out (exciuded) of the Road Map Project schooi districts. See appendix A on the 
caicuiation of dropout rates. 

Source: Authors' anaiysis of Washington Office of Superintendent of Pubiic instruction and Road Map Project 
district data, 2004/05-2012/13. 


graduation rate. Within this group long-term English proficient students stood out as 
having the highest four-year graduation rate of all student groups (76.4 percent). Recent- 
ly reclassified students graduated at a higher rate (69.4 percent) than did current English 
learner students but still trailed never-English learner students by 4-1 percentage points. 

After five years the graduation rate for current English learner students remained lower 
than that of other groups; however, the percentage point increases in graduation rate 
from year 4 to year 5 were largest for these students. Eor example, the four-year graduation 
rate of newcomers was 30.9 percentage points lower than that of never-English learner 
students, whereas the five-year graduation rate for this group was 20.0 percentage points 
lower. Eormer English learner students also saw increased graduation rates after five years, 
although the increases were more in line with those of never-English learner students. 

Dropout rates also differed substantially among subgroups of ever-English learner students 
(see table 2). Current English learner students were more likely to drop out than never- or 
former English learner students. Newcomers had the highest dropout rate of any group, at 
8.9 percent. The difference between the two subgroups of former English learner students 
was particularly stark: 8.8 percent of recently reclassified students dropped out (a rate com- 
parable with that of newcomers) compared with 3.1 percent of long-term proficient English 
learner students. 


After five years the 
graduation rate 
for current Engiish 
iearner students 
remained iower 
than that of other 
groups; however, 
the percentage 
point increases in 
graduation rate 
from year 4 to year 
5 were iargest for 
these students 


At the end of four years of high school 23.8 percent of all students had neither graduated 
nor dropped out (table 3). Of these students, 62.4 percent (1,422 of 2,279 students) enrolled 
in a fifth year of high school. The other 37.6 percent had unknown, missing, or no-show 
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Table 3. Number and percentage of students in six Washington state districts who 
neither graduated nor dropped out after four or five years of entering grade 9 in 
2008/09, by English learner status 


Student group 

Total 

number of 

Students who 
neither graduated 
nor dropped out 
after four years 

Students who 
enrolled in a fifth 
year of high school 

Students who 
neither graduated 
nor dropped out 
after five years 

or subgroup 

students 

Number 

Percent 

Number 

Percent 

Number 

Percent 

All students 

9,595 

2,279 

23.8 

1,422 

14.8 

1,162 

12.1 

Never-English learner 

6,943 

1,479 

21.3 

863 

12.4 

775 

11.2 

Ever-English learner 

2,652 

800 

30.2 

559 

21.1 

387 

14.6 

Current English learner 

1,333 

527 

39.5 

380 

28.5 

253 

19.0 

Newcomer 

604 

293 

48.5 

196 

32.5 

144 

23.8 

Established 

729 

234 

32.1 

184 

25.2 

109 

15.0 

Former English learner 

1,319 

273 

20.7 

179 

13.6 

134 

10.2 

Recently reclassified 

216 

47 

21.8 

33 

15.3 

22 

10.2 

Long-term proficient 

1,103 

226 

20.5 

146 

13.2 

112 

10.2 


Note: Students who neither graduated nor dropped out include students who are continuing their high school 
education and students whose outcomes after four or five years of high school are unknown (withdrawal code 
of missing, “unknown,” or “no show”). 

Source: Authors’ analysis of Washington Office of Superintendent of Public Instruction and Road Map Project 
district data, 2004/05-2012/13. 


withdrawal codes and could not be categorized as graduates or dropouts. The proportion 
of students who had neither graduated nor dropped out before the fifth year of high school 
was higher among ever-English learner students (30.2 percent) than among never-English 
learner students (21.3 percent). Among ever-English learner students, newcomers had the 
highest proportion of students who had neither graduated nor dropped out (48.5 percent), 
while long-term proficient English learner students had the lowest (20.5 percent). However, 
gaps between groups diminished by the end of the fifth year of high school, when 
23.8 percent of newcomers and 10.2 percent of long-term English proficient students had 
neither graduated nor dropped out (see table 3). 


The Road Map Project early warning indicators were unable to accurately identify many future 
dropouts across the six districts, especially among newcomer English learner students 


Determining the percentage of students who were flagged by indicator 1 (six or more 
absences plus at least one course failure in grade 9), indicator 2 (at least one suspension 
or expulsion in grade 9), or a composite (triggering either one or both indicators) pro- 
vides useful context for understanding the accuracy of the indicators (table 4). Overall, 
23.8 percent of students triggered one or both indicators, with ever-English learner stu- 
dents triggering them only slightly more than never-English learner students. Similar 
percentages of current (25.0 percent) and former English learner students (23.4 percent) 
triggered one or both indicators. These percentages are all substantially higher than the 
percentage of students who dropped out. 

There were notable differences among English learner subgroups in the percentage 
of students who triggered the composite indicator. Among current English learner stu- 
dents, newcomers triggered either one or both indicators only 13.9 percent of the time, 
even though they had the highest percentage of dropouts. In contrast, 34.2 percent of 


Overall, 

23.8 percent of 
students triggered 
one or both early 
warning indicators, 
with ever-English 
learner students 
triggering them 
only slightly more 
than never-English 
learner students 
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Table 4. Percentage of students in six Washington state districts who triggered 
each early warning indicator, for all students and English learner student subgroups 


who entered grade 9 in 2008/09 

student group 
or subgroup 

Triggered 
indicator 1 
(six or more 
absences pius at 
ieast one course 
faiiure in grade 9) 

Triggered 
indicator 2 
(at ieast one 
suspension 
or expulsion 
in grade 9) 

Triggered 
either one or 
both indicators 
(composite 
indicator) 

Percentage of 
students who 
dropped out 

All students 

20.3 

8.0 

23.8 

5.4 

Never-English learner 

20.1 

8.1 

23.6 

5.2 

Ever-Engllsh learner 

20.7 

8.0 

24.2 

5.9 

Current English learner 

21.1 

8.3 

25.0 

7.7 

Newcomer 

11.4 

4.5 

13.9 

8.9 

Established 

29.1 

11.5 

34.2 

6.7 

Former English learner 

20.4 

7.6 

23.4 

4.0 

Recently reclassified 

27.3 

12.0 

31.9 

8.8 

Long-term proficient 

19.0 

6.7 

21.7 

3.1 


Source: Authors' analysis of Washington Office of Superintendent of Public Instruction and Road Map Project 
district data, 2004/05-2012/13. 


established English learner students were flagged. Among former English learner students, 
21.7 percent of long-term proficient students triggered either one or both indicators, com- 
pared with 31.9 percent of recently reclassified students. 


For all student 
groups It was more 
common to trigger 
early warning 
indicator 1 (six or 
more absences plus 
at least one course 
failure in grade 9) 
than early warning 
indicator 2 (at least 
one suspension 
or expulsion 
in grade 9) 


Eor all student groups it was more common to trigger indicator 1 than indicator 2. The 
two groups with the highest rate of triggering any indicator — established and recently 
reclassified English learner students — were also the two groups that most often triggered 
indicator 2. 


It is important to know not only what percentage of students the indicators flag, but also 
whether the indicators can distinguish between students who are likely to drop out and 
those who are not. One way to address this question is to look at odds ratios (table 5; see 
appendix A for a discussion of odds ratios). Eor example, among never-English learner stu- 
dents, an odds ratio of 3.2 means that the odds of dropping out for a never-English learner 
student who triggered an indicator were 3.2 times higher than the odds of dropping out 
for never-English learner students who did not trigger the indicator. The odds ratios were 
statistically significant for all groups and subgroups except for recently reclassified English 
learner students. 


If an indicator had perfect precision, all students identified by that indicator in grade 9 
would drop out. However, the precision was low for a composite indictor based on trig- 
gering either indicator or both indicators. Overall, only 10.3 percent of students who 
were identified dropped out (see table 5). The indicators were most precise for new- 
comers (15.5 percent) and least precise for long-term proficient English learner students 
(6.7 percent). 

An indicator with perfect sensitivity would identify 100 percent of dropouts as being at risk 
when they were in grade 9. The sensitivity of the composite indicator (either indicator or 
both) was 45.0 percent for all students. This means that 55.0 percent of eventual dropouts 
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Table 5. Odds ratios, precision, and sensitivity of the composite dropout indicator 
for students in six Washington state districts who entered grade 9 in 2008/09, by 
English learner status 


Student group 
or subgroup 

Odds ratio 

Precision 

(percentage of students 
identified in grade 9 as 
at risk who dropped out) 

Sensitivity 
(percentage of 
dropouts identified in 
grade 9 as at risk) 

All students 

2 3*** 

10.3 

45.0 

Never-English learner 

2 2*** 

10.7 

48.1 

Ever-English learner 

2 Q*** 

9.2 

37.8 

Current English learner 

1.8** 

11.1 

35.9 

Newcomer 

2.1* 

15.5 

24.1 

Established 

1.9* 

9.6 

49.0 

Former English learner 

2.4** 

7.1 

41.5 

Recently reclassified 

1.0 

8.7 

31.6 

Long-term proficient 

2 4*** 

6.7 

47.1 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Source: Authors' anaiysis of Washington Office of Superintendent of Pubiic instruction and Road Map Project 
district data, 2004/05-2012/13. 


were not flagged by the indicators. Sensitivity was higher for never-English learner students 
(48.1 percent) than for ever-English learner students (37.8 percent). It was much lower for 
newcomers: only 24-1 percent of newcomer dropouts were correctly identified as at risk in 
grade 9. Sensitivity was highest for established English learner students (49.0 percent). Eor 
additional analyses of the accuracy of the indicators, see appendix B. 

Implications of the study findings 


Since passage of the No Child Left Behind Act of 2001, it has become common to consid- 
er the performance of current English learner students compared with that of non-English 
learner students, both in achievement on state assessments and in graduation rates. It is 
less common to examine findings for current and former English learner students separate- 
ly. This study examined the graduation and dropout rates not only for current and former 
English learner students, but also for subgroups within each of those categories. 

The study highlighted the much lower graduation rate for newcomers, compared not only 
with never-English learner students but also with established English learner students 
(other current English learner students who had first attended U.S. schools in grade 6 or 
earlier). The study found that the higher graduation rates of former English learner stu- 
dents were due to the particularly high graduation rates of long-term proficient students; 
recently reclassified students graduated at lower rates than never-English learner students. 
Within this sample of ever-English learner students, dropout rates were particularly high 
for newcomers and recently reclassified students. This underscores the utility of examining 
data for subgroups of English learner students separately. These findings may raise districts’ 
awareness of subgroups of English learner students who may be in particular need of addi- 
tional supports to graduate. 

The data also reveal that while a fifth year of high school boosted graduation rates for 
all students, the gains from a fifth year were largest for current English learner students. 


The higher 
graduation rates 
of former English 
learner students 
were due to 
the particularly 
high graduation 
rates of long- 
term proficient 
students; recently 
reclassified 
students graduated 
at lower rates than 
never-English 
learner students 
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particularly for newcomers. Providing the opportunity for a fifth year — and making a fifth 
year appealing and feasible to students — may be one way to support newcomers. 

The purpose of early warning indicators is to systematically identify students in grade 9 
who are at risk of dropping out, so that they can receive additional supports to improve 
their chances of graduating. However, selecting the best indicators can be challenging 
since tradeoffs need to be considered. A useful indicator is one that is precise enough not 
to flag too many students who would have graduated without additional supports, yet sensh 
tive enough to identify a substantial percentage of students who need additional intervem 
tions to avoid dropping out. 

One recent guide recommends that districts examine multiple indicators and select those 
with at least 50 percent precision and preferably closer to 67 percent (Jobs for the Future, 
2014). Ideally, the indicator would also be sensitive enough to identify at least 50 percent of 
eventual dropouts as being at risk in grade 9. However, because the most precise indicators 
are often the least sensitive, districts must make choices based on available resources and 
the number of students they hope to identify and serve. 

A review of early warning indicators summarized the precision and sensitivity of 110 
dropout indicators used in different settings over the past 30 years and revealed vast varn 
ation in precision and sensitivity (Bowers, Sprott, & Taff, 2013). Precision was at least 
50 percent for 35 of the 110 indicators (32 percent). Sensitivity was also at least 50 percent 
for 13 of those 35 indicators. In other words, only 12 percent of the 110 indicators met 
the recommendations of the guide (Jobs for the Future, 2014). However, only 7 indica- 
tors (6 percent) had precision as low as that of the Road Map Project indicators (only 
10.3 percent of all identified students dropped out). Of the 110 studies reviewed, 47 
(43 percent) had sensitivity values above 45 percent — the sensitivity of the Road Map 
Project composite indicator. Given that there were many examples with better precision 
and sensitivity, it seems likely that the Road Map Project could identify stronger indicators 
than those currently being used. 

Further research using data from the Road Map Project districts might identify other 
promising variables or outpoints. For example, many early warning indicators use a high 
number of absences as an indicator (Bruce, Bridgeland, Fox, & Balfanz, 2011). Increasing 
the number of absences in indicator 1 might improve its precision, although it might also 
reduce its sensitivity. Some districts also use transitions — such as moving into a district 
after grade 5 or making multiple moves in and out of the district — in their early warning 
indicators (Hauser & Koenig, 2011). Academic achievement, grade retention, and course 
performance in grade 9 may also be better indicators of dropout risk for English learner 
students (Gwynne, Pareja, Ehrlich, & Allensworth, 2012; Kim, 2011). 

Indicators also perform differently for different subgroups of ever-English learner students 
(Gwynne et al., 2012). Most troublesome, they flag few newcomers — the subgroup that is 
most at risk of dropping out. This could be because of the unique circumstances of new- 
comers, such as difficulty learning English quickly enough to be able to earn high school 
credits in content areas required for graduation, enrollment in electives rather than in core 
courses, or the interruption of their education because of war or time in refugee camps 
(Kanno, 2015). At the same time, the indicators identified many students from the other 
subgroups as at risk who were not actually at risk of dropping out. 


Given that there 
are many examples 
of early warning 
indicators with 
better precision 
and sensitivity, it 
seems likely that 
the Road Map 
Project could 
identify stronger 
indicators than 
those currently 
being used 
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Because the risk of dropping out is especially high for newcomers, and yet the indicators 
are least sensitive for these students, other schools and districts serving newcomers may 
want to examine the accuracy of their own indicators for this subgroup of students. If they 
also find that the sensitivity of the indicators is low, they may want to consider other ways 
of looking at the academic records of this group. For example, many districts use credit 
accumulation to monitor whether students are on track to graduate or at risk of dropping 
out. Schools and districts serving newcomer English learner students, in particular, may 
want to review the programs or supports they have in place to assist these students and to 
increase their graduation rates. 

Ultimately, the Road Map Project districts and other state and local education agencies 
and communities that use early warning indicators may want to regularly evaluate the 
accuracy of their indicators. As student characteristics, instruction methods, and inter- 
ventions change over time, the ability of indicators to correctly identify students who need 
additional services may also change. 


>ts student 
characteristics, 
instruction 
methods, and 
interventions 
change over 
time, the ability 
of indicators to 
correctly identify 
students who need 
additional services 
may also change 


Limitations of the study 


The findings of this study should be interpreted with caution because of limitations in 
the state and district data used in the analyses. First, there were challenges in accurately 
identifying some long-term proficient English learner students. Having incomplete data 
for English learner status before 2004/05 means that, for some students, it was not clear 
whether they had been English learner students before grade 4, which would put them in 
the long-term proficient subgroup. To better capture students in this subgroup, the study 
coded as long-term proficient all students with a primary language other than English or 
who spoke a language other than English at home. This allowed the identification of stu- 
dents who were likely to be English learner students, but it may have included a few stu- 
dents who were bilingual but so fully proficient in English by the time they entered school 
that they were never classified as English learner students. 

Second, the English learner status of students who transferred into a Road Map Project 
district after grade 9 was difficult to determine. Students who listed a home or primary 
language other than English who were not classified as English learners were categorized as 
long-term proficient English learner students even though they might have been recently 
reclassified. A few students transferring into Road Map Project districts after grade 9 (that 
is, students who were present in grade 8 or earlier but not in grade 9) could be catego- 
rized as established or recently reclassified English learner students. Less than 3 percent 
of students categorized as established or recently reclassified English learner students had 
transferred into a Road Map Project district after grade 9, compared with 36.4 percent of 
students categorized as newcomers, 10.8 percent of long-term proficient English learner 
students, and 10.8 percent of never-English learner students. 

Third, it was not possible to identify the graduation or dropout status for all students in 
the dataset. Specifically, 12.1 percent of students in the sample had a final withdrawal code 
that was missing, “unknown,” or “no show” after five years of high school. These students 
are included in the denominator for calculating graduation and dropout rates but not in 
the numerator because it could not be confirmed whether they graduated, continued in 
school, or dropped out. 
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Finally, although the data for the study came from a school year before the indicators were 
adopted for use across multiple districts and before the districts had formal early warning 
systems, it is possible that some struggling students who would have been flagged by an 
indicator received supportive interventions from their schools. If that were the case, and 
the interventions were effective, some flagged students who might have dropped out would 
instead have graduated. This could contribute to the finding that substantially more stm 
dents triggered an indicator than dropped out. Having data on the specific interventions 
students received might help explain the overall low precision of the indicators and add 
useful evidence for school and district leaders to think about when designing and improv- 
ing early warning systems. 
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Appendix A. Data and methodology 


This appendix defines key variables used in the study in greater detail and explains the 
analytic methods that were employed to produce the study’s results. 

Defining subgroups of Engiish iearner students 

Federal law requires that students who arrive at school speaking a home language other 
than English be tested to determine their level of proficiency in English. If students are 
not proficient, they are classified as current English learner students. Once a year, their 
proficiency is reassessed. Depending on assessment results and local requirements, students 
remain classified as current English learner students or are reclassified as former English 
learner students. 

This process is similar across the United States, but the requirements for reclassification 
vary in different states. In Washington state English learner classification is determined 
solely by an English learner student’s score on a single language assessment: the Wash- 
ington Language Proficiency Test until 2011/12 and the Washington English Language 
Proficiency Assessment starting in 2012/13. Both assessments tested students’ listening, 
speaking, reading, and writing in English. Both assessments also required students to score 
at level 4 (the highest level) overall to be reclassified as former English learner students. 
Students with an overall score of level 3 or below were categorized as current English 
learner students. English learner students continued to take a language proficiency assess- 
ment annually until reclassification. Once reclassified, former English learner students 
no longer took that assessment and could not be moved back into the group of current 
English learner students. 

Given the heterogeneity of English learner students, it can be challenging to determine 
the appropriate way to disaggregate them into subgroups. Sometimes home language 
or immigrant/U.S. born are the most relevant categories; ultimately, any categorization 
scheme must depend on the purpose of a given study. A prior study of early warning indi- 
cators and English learner students in Chicago influenced the definition of English learner 
student subgroups in this work (Gwynne et al., 2012). In that study, researchers identified 
grade 9 students as current or former English learner students and distinguished within 
those two groups between students who had been in those categories before grade 6 and 
those who had not. In this study of the Road Map Project districts, the structure and reli- 
ability of existing data altered the definitions of these categories somewhat. 

The four subgroups of ever-English learner students are described in the main report. This 
appendix provides additional details about the categorization of students — first for those 
who entered grade 9 in a Road Map Project district and then for those who transferred 
into a Road Map Project high school during grades 10-12 — and some limitations of the 
data used for the study. 

For this study, current English learner students (in grade 9) are subdivided into two sub- 
groups: newcomers and established English learner students. 

• Newcomer English learner students first enrolled in any Washington state school as 
an English learner student in grades 7, 8, or 9. Although most of these students 
were new arrivals from other countries, it is possible that some students had spent 
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time in another state before arriving in Washington state, and a few might have 
been born in the United States. For example, a MexicamAmerican child might 
have been born in Idaho and moved to Washington state in grade 7 but still be 
classified as an English learner student. The data provided no way to distinguish 
this child from a child who was born in Mexico and arrived in the United States, 
in Washington state, in grade 7. Therefore, it is possible that some students cat- 
egorized as newcomers could have been established English learner students in 
another state. Nonetheless, all English learner students who arrived in Washing- 
ton state in grades 7, 8, or 9 are defined as newcomers in this study with the under- 
standing that this might include some established English learner students who 
had arrived earlier in another state. 

• Established English learner students were those who were current English learner 
students in grade 6 and were still current English learner students in grade 9. Some 
of these students may have arrived in grades K-4 as English learner students, so 
that grade 9 could represent their sixth, seventh, eighth, ninth, or tenth year as 
a current English learner student. These students are often referred to as long- 
term English learner students in the research literature, which typically refers to 
students who have been classified as English learners for more than five years (for 
example, Kim & Herman, 2009). 

There are also two distinct subgroups of former English learner students (those who had 

been classified as English learner students but were later reclassified as English proficient): 

• Recently reclassified students were classified as current English learner students at 
the start of grade 7 but were reclassified as former English learner students by the 
start of grade 9. 

• Long-term proficient students were reclassified as former English learner students 
in grade 6 or earlier. This long-term proficient group is an imperfect category. An 
ever-English learner student in grade 9 in 2008/09 would have needed to attain 
proficiency in English by 2005/06 to be considered long-term proficient. Data 
on the English learner status of students who were reclassified prior to 2004/05 
were largely unavailable. To best estimate the number of students in this catego- 
ry, the study team included all students whose parents indicated that they had a 
primary language other than English or who themselves spoke a language other 
than English in the home. This may include a few never-English learner students 
who were bilingual from the time they started kindergarten. Relying solely on 
data codes for English learner status from 2004/05 would mean losing all former 
English learner students who were reclassified between kindergarten and grade 4. 
Given recent findings that English learner students are reclassified in 3.8 years 
on average, it made more sense to find a way to include the many students who 
are reclassified in grades 2, 3, and 4 in the analyses, even if it meant including a 
few never-English learner students who were either bilingual or whose parents 
spoke another language at home but did not teach it to their children (Greenberg 
Motamedi, 2015). To ensure the validity of this method, the study team checked 
the overlap between the two groups for later years that have more complete data 
and found that approximately 94 percent of students identified by their home lan- 
guage or their parents’ primary language were also English learner students in the 
primary grades. 
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Determining graduation and dropout rates 


Students were considered founyear graduates if their file included a code for graduating with 
a regular high school diploma, an associate’s degree, or with an Individualized Education 
Program completion diploma (modified diploma) in four or fewer years after entering grade 
9. To calculate the adjusted graduation rate, the number of graduates was divided by the 
number of incoming students in grade 9, correcting for the number of students transferring 
in or out of the six Road Map Project districts, using the following formula: 


Graduation Rate = 


# of students who graduated within four years of entering grade 9 
# of students entering grade 9 in 2008109 + transfers in - transfers out 


Adding in the number of students earning a diploma, associate’s degree, or modified 
diploma within five years of starting high school permitted the calculation of five-year 
graduation rates. 

Similarly, Washington state’s Office of Superintendent of Public Instruction uses the fol- 
lowing formula to calculate four-year adjusted cohort dropout rates, or the percentage of 
students in any grade 9 high school class who drop out within four years of entering grade 
9: 


Dropout Rate = 


# of students who dropped out within four years of entering grade 9 
# of students entering grade 9 in 2008109 + transfers in - transfers out 


The number of students who dropped out includes students with withdrawal codes of 
“dropout” or “General Educational Development (GED)” for the final year they were in the 
dataset. 


The denominators for four- and five-year graduation rates and for dropout rate are the 
same. Students who transferred out of the six Road Map Project districts at any point after 
grade 9 are neither included in the study sample nor in the denominators for calculating 
graduation and dropout rates. 

Students who did not have a withdrawal code that indicated they dropped out or graduat- 
ed were handled in the following ways: 

• Students who transferred out of the Road Map Project districts after entering grade 
9 (1,849 students or 16.1 percent of students who entered grade 9 in 2008/09) were 
not included in the calculation of graduation and dropout rates or in the study 
sample. About 10 percent of these students who transferred out of the Road Map 
Project districts did so after their fourth year of high school. Students who trans- 
ferred out are not included in the study sample or in the calculation of graduation 
and dropout rates because the Road Map Project districts do not have influence 
over these students’ outcomes after four or five years of high school after they leave. 

• Students who did not live to graduate within five years — less than 0.1 percent of 
the sample of students who entered grade 9 in 2008/09 — were not included in the 
calculation of graduation and dropout rates or in the study sample. 

• Students whose last withdrawal code was “unknown” or “no show” (845 students 
or 8.8 percent of students in the study sample) are included in the study sample and 
in denominators of both equations, but they are not included in the numerators 
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of either equation. An additional 3.3 percent of students in the study sample (317 
students) have a missing final withdrawal code. Some of these students may have 
continued their education in the following year, for which data were not avail' 
able (45.4 percent of students missing the last withdrawal code were enrolled in 
2012/13). Because it cannot be determined whether students who have withdrawal 
codes that are missing, “unknown,” or “no show” in the last year they were present 
in the data graduated, dropped out, transferred, or died, they are included in the 
sample but considered neither graduates nor dropouts. 

The graduation and dropout formulas yielded the percentages reported in tables 1 and 2 in 
the main report. 

Defining course faiiures 

Students were considered to have a course failure in grade 9 if any course they enrolled 
in had a grade of F, E, unsatisfactory, no pass, no credit, or an incomplete that was never 
replaced by a passing grade. Grade notations of withdraw, in progress (but never resolved 
as a grade), and N/A were not coded as course failures. 

Note that the indicators for the Road Map Project include failures in any course subject, 
including electives. This differs from early warning indicators in some other settings, such 
as Chicago, where only failures in core subject areas are used to flag a student as at risk 
(Allensworth & Easton, 2007). 

Analyzing the accuracy of early warning indicators 

The logistic regression model and corresponding predictive probabilities were used to 
understand the ability of the two Road Map Project early warning indicators to accurately 
predict the probability of dropout. This approach has been employed previously to examine 
whether widely used indicators of risk are effective predictors of students likely to drop out 
and which indicators are most predictive (Allensworth & Easton 2007; Balfanz, Wang, & 
Byrnes, 2010; Gleason & Dynarski, 2002). 

The model can be expressed as: 

ln[Ml -P)], = ^0 + PilNDfCATOR. + s. 

where In is the natural logarithm, where exp = 2.71828..., p is the probability that 
the event of dropout occurs, p(dropout = I), p/(I - p) is the “odds” and ln[p/(I - p) is the 
log odds, or “logit.” The model was estimated separately for indicator 1, indicator 2, and a 
composite indicator (triggering either one or both indicators). Each model was estimated 
separately for each English learner student subgroup and for never-English learner stm 
dents. In the indicator 1 model the dichotomous dependent variable equals 1 if the student 
dropped out and 0 if the student did not drop out. The independent variable INDICATOR 
is 1 if a student has six or more absences plus at least one course failure in grade 9 and 0 
otherwise. Similarly, for the indicator 2 model the independent variable INDICATOR is 
1 if a student has a suspension or expulsion in grade 9 and 0 otherwise. In a third set of 
analyses the independent variable INDICATOR is 1 if a student met the criteria for trig- 
gering either one or both indicators and was 0 otherwise. 
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Stata, the statistical software used for analysis, produces several indices of classification 
accuracy to evaluate results from logistic regressions upon running a postestimation 
command “estat classification.” In this study, the students are categorized into one of the 
four categories (table Al). 

Once students are grouped into one of the four categories, it becomes possible to calculate 
four indices. The first index, precision, is the percentage of students who drop out among 
all students who are identified as at risk of dropping out by the indicator — or the number 
of true positives divided by the sum of true positives and false positives (A/[A+B]). To a 
certain degree, precision is the accuracy of the early warning flag. However, if schools use 
the early warning flag to provide helpful interventions to students, fewer flagged students 
may drop out, which would reduce precision. Higher precision implies the indicator was 
able to minimize false positives — that is, not overidentify students so that students not at 
risk of dropping out do not have to undergo interventions. 

The second index, sensitivity, is the percentage of students who are identified as at risk of 
dropping out among all students who drop out — or the number of true positives divided by 
the sum of true positives and false negatives (A/[A+C]). It provides information about how 
well the early warning indicators identify dropouts. 

The third index, specificity, is the percentage of students who are identified as not at risk 
of dropping out among all students who do not drop out — or the number of true negatives 
divided by the sum of true negatives and false positives (D/[D+B]). This provides informa- 
tion about the percentage of students who did not trigger early warning indicators who 
were genuinely not at risk. 

Finally, some researchers also consider negative predictive power, the percentage of stm 
dents who do not drop out among all students who are identified as at risk by the indicator. 
This can be calculated as the number of true negatives divided by the sum of false nega- 
tives and true negatives (D/[C+D]). High negative predictive power implies the indicator 
was able to minimize false negatives — that is, it avoids under-identifying students so that 
more students truly at risk of dropping out can receive timely interventions. 


Table Al. Example of classification of results from logistic regression 


Actual outcome 

Prediction based on indicators 

Dropped out 

Did not drop out 

At risk 

A. Accurate: True positive 

B. Inaccurate: False positive 

Not at risk 

C. Inaccurate: False negative 

D. Accurate: True negative 


Source: Authors’ illustration. 




Appendix B. Supplemental results 


This appendix provides more detail about the strength and significance, as well as the pre- 
cision, sensitivity, specificity, and negative predictive power of the early warning indicators 
used across the Road Map Project districts. 

Odds ratios provide information about the likelihood of an outcome occurring for one 
group compared with another. Logistic regression generated the odds ratios displayed in 
table Bl. The regression model was estimated separately for indicator 1, indicator 2, and a 
composite indicator (triggering either indicator or both indicators). Furthermore, models 
were estimated separately for all students, never-English learner students, and each English 
learner student subgroup. 

The statistical significance of some odds ratios in table Bl means that triggering an indicator 
yields statistically different odds of dropping out. For example, among never-English learner 
students, those who triggered indicator 1 in grade 9 were statistically significantly more likely 
to drop out than those who did not. The odds ratio of 3.0 means that the odds of dropping 
out for never-English learner students who triggered indicator 1 were 3.0 times higher than 
the odds of dropping out for never-English learner students who did not trigger the indicator. 

The odds ratio was higher on average for indicator 2 than for indicator 1 or for the com- 
posite indicator (triggering one indicator or both). However, the composite indicator was 
statistically significant more often than either indicator on its own. 

The analysis software, Stata, generated indices of classification accuracy for the logistic 
regression results: precision, sensitivity, specificity, and negative predictive power. 


Precision. The precision of indicator 1 for never-English learner students was 10.7 percent 
and for ever-English learner students 8.7 percent (table B2). There was variation among 


Table Bl. Odds ratios for early warning indicators as predictors of dropping out for 
students from six Washington state districts who entered grade 9 in 2008/09, by 
English learner status 

student group or subgroup 

Indicator 1 

(six or more absences 
pius at ieast one course 
faiiure in grade 9) 

indicator 2 

(at ieast one suspension 
or expuision in grade 9) 

Composite indicator 
(triggering either one 
or both indicators) 

All students 

2 0*** 

2 9*** 

2 3*** 

Never-English learner 


0 0j(c5(:4: 

0 2*** 

Ever-Engllsh learner 

2 

2 0** 

2 Q*** 

Current English learner 

1.6* 

1.5 

1.8** 

Newcomer 

1.6 

2.4 

2.1* 

Established 

1.9* 

1.3 

1.9* 

Former English learner 

2 

3 5*** 

2 4** 

Recently reclassified 

0.9 

0.8 

1.0 

Long-term proficient 

2 7** 

5.6*** 

0 4**=(c 

* Significant at p < .05; ** significant at p < .01; *** significant at p <.001. 


Source: Authors’ analysis of Washington Office of Superintendent of Public Instruction and Road Map Project 
district data, 2004/05-2012/13. 
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Table B2. Summary of classification accuracy results by Indicator and overali for students from six 
Washington state districts who entered grade 9 in 2008/09, by English iearner status 


student group or subgroup 

Precision 
(percentage 
of students 
identified in grade 

9 as at risk of 
dropping out who 
did drop out) 

Sensitivity 
(percentage of 
students who 
dropped out who 
were identified in 
grade 9 as at risk 
of dropping out) 

Specificity 
(percentage of 
students who 
did not drop 
out who were 
identified in grade 

9 as not at risk 
of dropping out) 

Negative 
predictive power 
(percentage of 
students identified 
in grade 9 as not 
at risk of dropping 
out who did 
not drop out) 

Correctiy 
ciassified 
(percentage of 
students who 
were correctiy 
ciassified as at 
risk or not at risk 
of dropping out) 

Indicator 1: Six or more absences plus at least one course failure in grade 9 

All students 

10.1 

37.9 

80.8 

95.8 

78.4 

Never-English learner 

10.7 

40.9 

81.1 

96.1 

79.0 

Ever-English learner 

8.7 

30.8 

79.9 

94.9 

77.0 

Current English learner 

10.7 

29.1 

79.6 

93.1 

75.7 

Newcomer 

13.0 

16.7 

89.1 

91.6 

82.6 

Established 

9.9 

42.9 

71.9 

94.6 

70.0 

Former English learner 

6.7 

34.0 

80.2 

96.7 

78.3 

Recently reclassified 

8.9 

26.3 

72.6 

91.1 

68.5 

Long-term proficient 

6.2 

38.2 

81.6 

97.6 

80.2 

Indicator 2: At least one suspension or expulsion in grade 9 

All students 

12.7 

18.8 

92.6 

95.2 

88.6 

Never-English learner 

13.4 

20.6 

92.6 

95.5 

88.9 

Ever-English learner 

10.9 

14.7 

92.5 

94.6 

87.9 

Current English learner 

10.8 

11.7 

92.0 

92.6 

85.7 

Newcomer 

18.5 

9.3 

96.0 

91.5 

88.2 

Established 

8.3 

14.3 

88.7 

93.5 

83.7 

Former English learner 

11.0 

20.8 

93.0 

96.6 

90.1 

Recently reclassified 

7.7 

10.5 

87.8 

91.1 

81.0 

Long-term proficient 

12.2 

26.5 

93.9 

97.6 

91.8 

Composite indicator (triggering either one or both indicators) 

All students 

10.3 

45.0 

77.5 

96.1 

75.7 

Never-English learner 

10.7 

48.1 

77.8 

96.4 

76.2 

Ever-English learner 

9.2 

37.8 

76.7 

95.2 

74.4 

Current English learner 

11.1 

35.9 

75.9 

93.4 

72.8 

Newcomer 

15.5 

24.1 

87.1 

92.1 

81.5 

Established 

9.6 

49.0 

66.9 

94.8 

65.7 

Former English learner 

7.1 

41.5 

77.4 

96.9 

76.0 

Recently reclassified 

8.7 

31.6 

68.0 

91.2 

64.8 

Long-term proficient 

6.7 

47.1 

79.1 

97.9 

78.2 

Source: Authors’ analysis of Washington Office of Superintendent of Public Instruction and Road Map Project district data, 
2004/05-2012/13. 


English learner subgroups, with long-term proficient English learner students having the 
lowest precision for indicator 1 (6.2 percent) and newcomers the highest (13.0 percent). 

Patterns in the precision of indicator 2 were somewhat similar: There was a 2.5 percent- 
age point difference between never- (13.4 percent) and ever-English learner students 
(10.9 percent) and the precision of indicator 2 was highest for newcomers (18.5 percent). 
However, for this indicator, the lowest level of precision was for recently reclassified English 
learner students (7.7 percent). 
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The precision for the composite indicator ranged from a low of 6.7 percent for long-term 
proficient English learner students to a high of 15.5 percent for newcomers. This means 
that the indicators accurately flagged future dropouts between 6.7 and 15.5 percent of the 
time. Precision did not increase for the composite indicator. Indicator 2 alone had slightly 
better precision than the composite indicator for all subgroups except for established and 
recently reclassified English learner students. Of course, precision statistics cannot reveal 
whether having been flagged and receiving interventions prevented some potential drop- 
outs from dropping out. 

Sensitivity. Overall, the use of indicator 1 in grade 9 correctly identified 37.9 percent of 
all eventual dropouts, but the proportion was higher for never-English learner students 
(40.9 percent) than for ever-English learner students (30.8 percent). The sensitivity of 
indicator 1 looked very different for the subgroups of ever-English learner students: as low 
as 16.7 percent of newcomers and as high as 42.9 percent of established English learner stu- 
dents. Eor indicator 2, a generally similar pattern in sensitivity was evident: it was higher 
for never-English learner students (20.6 percent) than for ever-English learner students 
(14.7 percent), and there was substantial variation among English learner subgroups. The 
sensitivity of indicator 2 was lowest for newcomers (9.3 percent) and highest in this case 
for long-term proficient students (26.5 percent). 

When the composite indicator (triggering either indicator or both) was used, sensitivity 
increased by about 5 to 9 percentage points compared with indicator 1 alone and sub- 
stantially more compared with indicator 2. The general patterns observed with indica- 
tor 1 are also evident for the composite indicator: Sensitivity is higher for never-English 
learner students than for ever-English learner students. Among subgroups, sensitivity is 
lowest for newcomers (24.1 percent) and highest for established English learner students 
(49.0 percent). This means that the indicators failed to identify more than half of all even- 
tual dropouts, from any subgroup, and missed 75.9 percent of newcomers who dropped out. 

Precision and sensitivity are useful ways of examining how well indicators identify students 
who will later drop out. At the same time, districts want to ensure that they are accurate 
in identifying students who will not drop out. They do not want to inaccurately flag these 
students as being at risk and provide expensive interventions if the students are not at risk, 
and yet they also want to avoid incorrectly identifying them as not at risk when in fact 
they later drop out. These concerns are addressed by examining specificity and negative 
predictive power. 

Specificity. Most students neither triggered an indicator nor dropped out. Some 
23.8 percent of students triggered an indicator, meaning 76.2 did not (see table 4 in the 
main report). Similarly, 5.4 percent of students dropped out, meaning 94.6 percent did 
not drop out (see table 2 in the main report). Eor that reason, specificity values — which 
provide information on the accurate identification of nondropouts — were much higher 
than precision or sensitivity values. 

It is easier to identify the many students who are not at risk. These values were similar for 
never- and ever-English learner students for indicator 1, indicator 2, and for the composite 
indicator. Eor example, for the composite indicator the rates are 77.8 percent for never-English 
learner students and 76.7 percent for ever-English learner students. Within subgroups, speci- 
ficity was lowest for established and recently reclassified English learner students. 
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Negative predictive power. Of all the students identified as not at risk of dropping out 
using the composite indicator, 96.1 percent did not drop out. Although negative predic- 
tive power was lowest for newcomer and recently reclassified English learner students, it 
remained high (92.1 and 91.2 percent, respectively). Similar patterns held for indicator 1 
and indicator 2. 

Finally, combining precision and negative predictive power yields the percentage of stm 
dents who were correctly classified: 75.7 percent all students, using the composite indicator, 
but only 64.8 percent of recently reclassified English learner students. While the percent 
correctly classified can be used as an overall summary measure of accuracy, it does not 
facilitate decisionmaking as accurately as the separate indicators — such as precision and 
sensitivity — because those indicators allow policymakers to consider the tradeoffs involved 
in making false positives and false negatives. For this reason, the percentage of students 
correctly classified is not discussed in the main report. 

Using the composite indicator improves the percentage of dropouts flagged by the early 
warning system (sensitivity) but it lowers specificity. Weighing higher sensitivity over 
higher specificity essentially means accepting the mistaken identification of some students 
as at risk who are not really at risk, in exchange for also identifying more students genuine- 
ly at risk. This would be the argument for using the composite indicator rather than only 
one indicator. 
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Making Connections 

Studies of correlational relationships 


Making an impact 

Studies of cause and effect 


What’s Happening 

Descriptions of policies, programs, implementation status, or data trends 


What’s Known 

Summaries of previous research 


Stated Briefiy 

Summaries of research findings for specific audiences 


Appiied Research Methods 

Research methods for educational settings 


Toois 
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